SVE 256-bit Implementation of mxfp4 gemv kernel - #28141
SVE 256-bit Implementation of mxfp4 gemv kernel#28141abhijain1204fujitsu wants to merge 3 commits into
Conversation
github-actions
Bot
added
the
ggml
|
Hi @abhijain1204fujitsu, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
github-actions
Bot
removed
the
draft
|
@ggerganov, @Alcpz |
Alcpz
left a comment
Alcpz
left a comment
There was a problem hiding this comment.
I've replicated the performance on a different machine, e2e instead of Neon vs SVE:
| Model | Test | Threads | master SVE (t/s) | PR SVE (t/s) | Diff (%) |
|---|---|---|---|---|---|
| gpt-oss-20b | PP128 | 4 | 31.62 | 35.44 | 12.1 |
| gpt-oss-20b | TG128 | 4 | 22.78 | 24.90 | 9.3 |
| gpt-oss-20b | PP128 | 8 | 56.55 | 60.97 | 7.8 |
| gpt-oss-20b | TG128 | 8 | 37.64 | 38.91 | 3.4 |
| gpt-oss-20b | PP128 | 16 | 104.69 | 110.26 | 5.3 |
| gpt-oss-20b | TG128 | 16 | 54.65 | 56.41 | 3.2 |
| Qwen3.6-35B-A3B | PP128 | 4 | 41.38 | 42.80 | 3.4 |
| Qwen3.6-35B-A3B | TG128 | 4 | 18.40 | 19.06 | 3.6 |
| Qwen3.6-35B-A3B | PP128 | 8 | 72.33 | 73.94 | 2.2 |
| Qwen3.6-35B-A3B | TG128 | 8 | 28.23 | 28.99 | 2.7 |
| Qwen3.6-35B-A3B | PP128 | 16 | 125.20 | 127.04 | 1.5 |
| Qwen3.6-35B-A3B | TG128 | 16 | 35.65 | 36.19 | 1.5 |
| LFM2.5-8B-A1B | PP128 | 4 | 81.10 | 90.30 | 11.3 |
| LFM2.5-8B-A1B | TG128 | 4 | 47.67 | 50.53 | 6.0 |
| LFM2.5-8B-A1B | PP128 | 8 | 143.14 | 153.41 | 7.2 |
| LFM2.5-8B-A1B | TG128 | 8 | 76.29 | 78.79 | 3.3 |
| LFM2.5-8B-A1B | PP128 | 16 | 259.69 | 271.01 | 4.4 |
| LFM2.5-8B-A1B | TG128 | 16 | 107.43 | 107.53 | 0.1 |
I observed worse performance E2E for Qwen, but that's because the NEON path is faster, not due to this PR.
Built and measured with:
cmake -DCMAKE_BUILD_TYPE=Release \
-DGGML_NATIVE=OFF \
-DGGML_CPU_ARM_ARCH=armv8.6-a+sve
build/bin/llama-bench -p 128 -n 128 -t 4,8,16Perplexity also checks out (added master as I saw differences in Perplexity, but those seem to come from other SVE kernels):
| Model | NEON | SVE (PR) | SVE (master) |
|---|---|---|---|
| gpt-oss-20b MXFP4 | 237.1192 ± 8.81243 | 238.5075 ± 8.86238 | 238.5075 ± 8.86238 |
| Qwen3.6-35B-A3B MXFP4_MOE | 5.5133 ± 0.14971 | 5.5160 ± 0.14968 | 5.5160 ± 0.14968 |
| LFM2.5-8B-A1B MXFP4_MOE | 27.2469 ± 1.11531 | 27.1599 ± 1.11147 | 27.1599 ± 1.11147 |
# Run as
llama-perplexity -f wikitext-2-raw/wiki.test.raw --chunks 30 -t 16
| UNUSED(blocklen); | ||
|
|
||
| #if defined(__aarch64__) && defined(__ARM_FEATURE_SVE) | ||
| if (svcntb() == 32) { |
There was a problem hiding this comment.
NIT: I've seen this if in other SVE kernels as svcntb() * 8 == 256. If those are yours, I'd push to have a consistent conditional logic
3 participants
Overview
This PR aims to add SVE 256 support for the ggml_gemv_mxfp4_4x4_q8_0 kernel found in profiling of gpt-oss-20b MoE model.
Additional information
GPT-OSS-20B SVE256 Performance
(Taken on Graviton3e)
Perplexity:
Requirements
Co Authors: @anubhavfujitsu